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Abstract 

The Traffic Light Report (TLR) project is an educational intervention 
designed for pharmacy undergraduates. This paper reports on analysis of 
TLR data specifically focusing on its potential as an innovative tool which 
combines Miller’s pyramid, technology and student voice to examine a 
curriculum for Assurance of Learning (AoL). In 2014, educators mapped 
each summative assessment to the relevant National Competency Standards 
Framework for Pharmacists in Australia (NCS) alongside levels of expected 
performance on Miller’s pyramid of clinical competence (Knows, Knows 
how, Shows how, Does). Simultaneously, students were invited to self-reflect 
using the same performance levels. The Miller’s scale enabled a comparison 
between students’ and their educators’ understanding of the performance 
level demanded by assessments. Analysis highlighted a disconnect between 
students ’ and their educators ’ interpretations of the same assessed 
curriculum. The TLR facilitates quality enhancement by providing educators 
and their students with a logical meeting point for discussing foundation, 
scaffolding and integration of assessment across a course for AoL. This has 
portability to other professional disciplines. 
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Introduction 

Higher education providers in Australia are under increasing pressure to integrate professional 
standards in the development and assessment of Course Learning Outcomes (Australian 
Government Department of Education and Training, 2015). The National Competency Standards 
Framework for Pharmacists in Australia (NCS) is the relevant professional standard for 
pharmacists. The NCS are grouped into eight Domains: (1) Professional and ethical practice; (2) 
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Communication, collaboration and self-management; (3) Leadership and management; (4) Review 
and supply prescribed medicines; (5) Prepare pharmaceutical products; (6) Deliver primary and 
preventative health care; (7) Promote and contribute to optimal use of medicines; and, (8) Critical 
analysis, research and education. The Australian Pharmacy Council who accredits the university 
courses highlight the importance of the NCS to pharmacy education by explaining that: 

Since the entry-level competencies are to be met at entry to professional practice, they 
can serve as a source of guidance to the teaching and learning expected. ...The goal of 
initial pharmacy education is to produce graduates with the requisite knowledge, skills 
and attributes for entry to an intern training program. 

(Australian Phannacy Council, 2012, p. 15) 

Pharmacist competence assures patient safety. Familiarity with the NCS should begin with first 
year undergraduates so they can engage with their profession’s competence continuum (Coombes, 
Bates, Duggan, & Galbraith, 2011) and continually monitor and improve their performance. 
Professional competency standards such as the NCS can be combined with a scale to communicate 
a logical continuum towards competence for educators and their students. Miller’s (1990) Pyramid 
of Clinical Competence which classifies competence along a continuum of “Knows,” “Knows 
how,” “Shows how” and Does” is used by pharmacists and other health professionals worldwide 
for this purpose (General Pharmaceutical Council, 2011; Kelley & Demb, 2006; Wass, Van der 
Vleuten, Shatzer, & Jones, 2001). Other frameworks exist, however Miller’s pyramid is a simple 
conceptual model that clearly communicates the early stages of the competence continuum. Each 
level on Miller’s pyramid represents increasing capability and increased integration of knowledge, 
skills and attributes. Prior to performing at a “Does” level in the practice setting with a client, an 
individual must have developed essential underpinning knowledge, skills and attributes. For 
optimal learning and assured practitioner competence, these stages are best scaffolded. In other 
words, graduates who have had their learning scaffolded and are provided a “solid foundation” 
having progressed through the “Knows” level (observation and basic knowledge recall), “Knows 
how” (understands and applies knowledge) and “Shows how” (integrates knowledge, skills, 
attitudes to perform in simulated settings) (Miller, 1990; Van Der Vleuten & Schuwirth, 2005; 
Wass, et al., 2001) are given the greatest chance of success. 

Familiarity with the NCS among phannacy students in Australia (Nash, Chalmers, Stupans, & 
Brown, 2015a) and at the university of interest was discovered to be poor (Nash, Chalmers, 
Stupans, Brown, 2016a). To address poor student familiarity with the NCS and develop students’ 
self-reflection skills, the Traffic Light Report (TLR) project was conceptualised. The TLR 
elements (NCS and Miller’s pyramid) were combined to ensure the usability of the NCS for 
educators and their students in their context. Influenced by the Triple Jump Test (Meszaros et ah, 
2009), the mileMarker (Szilagyi, 2008), topographical curriculum maps (Plaza, Draugalis, Slack, 
Skrepnek, & Sauer, 2007) and the work of Janke, Traynor and Sorensen (2011), the TLR applies a 
course wide approach to assure student learning. 

Essential to the TLR design is curriculum mapping. Curriculum mapping is commonly used in 
higher education to provide a course level summary of a curriculum, enable curriculum review and 
provide evidence for accreditation and quality enhancement requirements. Martone and Sireci 
(2009) suggested that the mapping process is more usefid than the results as it helps educators see 
how assessments can connect to learning activities. Oftentimes, the mapping process and 
discussion it prompts amongst educators (and students) becomes the most valuable component 
(Kelley, McAuley, Wallace, & Frank, 2008). Thus, a teaching team must design and map their 
course together (Lawson, 2014, 2015a) and assess their curriculum first for student learning and 
second in terms of its effectiveness to measure achievement (Boud & Falchikov, 2005). To support 
this, a range of mapping tools currently exist (Harden, 2001; Oliver, Ferns, Whelan, & Lilly, 2010) 
including, of particular relevance to the work reported here, Lawson’s Curriculum Design 
Workbench (Lawson, 2014, 2015a). The Curriculum Design Workbench is built on a philosophy 
of Assurance of Learning (AoL) (AACSB, 2013; Lawson, 2015b) which emphasises a teaching 
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team approach to course design to ensure student knowledge and skills are scaffolded and, 
importantly, educators and their students are aware of where this scaffolding takes place (Lawson, 
2015a; Lawson et al., 2015). In its most simplistic form, AoL is defined as “the process by which 
learning outcomes are measured against specific course goals” (Hall & Kro, 2006). 

This research used the data from the TLR to highlight that educators at the university of interest 
and their students describe a very different understanding of expected performance levels, as 
described through the Miller’s pyramid. This paper seeks to explore possible reasons for this 
difference. The findings from the TLR have implications for educators, students and regulators and 
have portability to other professions. 

Method 

This project was conducted at one Australian university over two semesters (Semesters 1 and 2, 
2014), where the full time pharmacy course is comprised of four units per semester. Each unit had 
2-8 summative assessment tasks. Experiential placements took place in Third and Fourth year, 
students in First year visited their allocated workplace briefly whilst Second year students did not 
have scheduled placements. The data presented on Miller’s pyramid in this paper is one part of the 
TLR research project, with other applications and outcomes having been described and reported 
elsewhere, including its ability to improve student knowledge of the NCS (Nash et ah, 2016a) and 
provide opportunities to develop self-assessment skills (Nash, Stupans, Chalmers, & Brown, 
2016b). 

A pragmatist methodological frame (Creswell, 2013), educational design (Van Den Akker, 
Gravemeijer, McKenney, & Nieveen, 2006) and action research philosophies (Creswell, 2013) 
informed the authors’ approach. Triangulation of data (Creswell, 2013) from student self¬ 
reflection, curriculum mapping data and feedback surveys made it possible to interpret the 
curriculum from the viewpoint of students and their educators. Data collection for each perspective 
is now described to aid the understanding of the analyses carried out. 


Educators’ perspective (assessed curriculum) 

For each assessment item in each unit of the course, educators entered details of the task into a 
customised Access database, the relevant NCS for each assessment, and the level of proficiency on 
Miller’s pyramid that students were expected to achieve (Figure 1). A map was then produced of 
all the relevant NCS and level of Miller’s proficiency for each assessment in the course. The 
authors interpreted this curriculum map as the educators’ perspective of the assessed curriculum. 
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Figure 1. Design method and origin of data for TLR and associated comparison 


Students’ perspective (learnt curriculum) 

At the conclusion of scheduled semester examinations, enrolled students were invited to self- 
reflect on their performance against the eight NCS on Miller’s pyramid via a 10 minute online 
survey (Figure 2). The self-reflection was open for two weeks. Prior to this project, students had 
limited exposure to the NCS which, as previously noted, consists of eight Domains, each 
containing multiple competency standards. Specifically students were asked to: consider your 
current performance level (Knows, Knows how, Shows how, Does) for each competency standard 
(NCS). The authors interpreted the results from this survey as the students’ “learnt” curriculum. 
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Domain 1. Professional A Ethical Practice 


* Domain 1. Professional and Ithkal Practice 


Consider your current performance level (Knows, Knows How, Shows, Does) for each Competency Standard (CS). 


Then for each of the CS use the traffk light system to indicate if you feel the CS requires further attention. 

e 


Performance Level: 

Know* (Knowledge eg Multiple Choice Questions) 

Knows How (Describes how eg Applies knowledge in written esse) 
Shows (Performance eg. Simulation, role play) 

Does (Act>on-real Me eg. Placements) 


Traffic Lightt 
Okay Green) 

May need attention ;• 

Urgent attention < ted) 


1.1 Practice legally 

1.2 Practice to accepted 
standards 

1.3 Deliver patient- 
centred care' 

1.4 Manage quakty & 
safety 

1.5 Maintain and extend 
professional competence 


Does 


Orange Red 


* Have any of the Competency Standard (CS) in Domain 1. Professional A Fthkal practice been assessed in your B.Pharm this Semester? 


Figure 2. Screenshot of the student self-reflection tool displaying how they reported their progress 
against the NCS and Miller’s pyramid on Domain 1. This was repeated for the 8 Domains. 


Perspectives combined and contrasted 

The educators’ perspective was represented by the curriculum mapping entered into the TLR. The 
students’ self-reflection provided the students’ perspective (Figure 1). 

This paper is focussed on the analyses of data associated with Miller’s pyramid. For the purpose of 
this analysis, data were extracted from the TLR and summarised by year and semester of 
enrolment, for example, Fourth year, Semester 1. The “instances” used in the analysis were 
defined as: 

(i) an individual student’s self-reflection on a given standard within the NCS, for 
example, “Practice to accepted standards” (Standard 1.1), at a “Does” level 
represents one instance; and, 

(ii) their educators’ mapping against a given standard for an assessment task. 

As each NCS may have been mapped to multiple summative assessments, the database was 
programmed to select the instance where the student performed at the highest level in assessment 
on Miller’s pyramid, for example, “Does” was higher than “Knows.” The educators’ mapping of 
the performance level of the summative assessment was directly compared with the Miller’s level 
selected by the students in their self-reflection to determine agreement that is, whether the two 
perspectives matched. Non-parametric data analyses were conducted using SPSS Version 22. 

These data were also extracted into the statistical program R to provide heat-maps that described 
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the curriculum at an individual Domain level of the NCS. A “hot spot” was where the ratio, 
indicating the perceived degree of focus of the curriculum on a particular NCS Domain at a given 
Miller’s level, was greater than 0.6. The following provides an example formula used to generate 
the ratios reported in the heat-maps: 

Ratio (Domain2KnowsYear2) = Count of Assessments on Domain 2 at Knows Level for 
all students in Year 2 / Count of Assessments on Domain 2 for All Millers Levels for all 
Year 2 students. 

This research was approved by the Social Sciences Human Research Ethics Committee (Tasmania 
Network); H00013591. The lead author handled all data and disseminated the TLR; to eliminate 
ethical concerns, she held no teaching responsibilities at the time. Student involvement was 
voluntary. Following receipt of their TLR, students were invited using an online survey to provide 
feedback on the TLR educational intervention. 

Results 

Students who participated in the self-reflection represented 42% (n= 69) of the invited students 
(A=163) in Semester 1 and 26% (n= 52) of invited students in Semester 2 (A=198). Table 1 
provides a breakdown of student participation by year level and by semester of participation. 
Students are referred to in this paper, by Year of course, Gender and an identifying number, for 
example, (Year 2, Malel9), and (Year 4, Female22). 

Table 1 

Student participation by semester/year level of course 



Year 1 

Year 2 

Year 3 

Year 4 

#=121 


n (%) 

n (%) 

n (%) 

n (%) 


Semester 1 

1 (1.45%) 

27(39.13%) 

21 (30.43%) 

20 (28.99%) 

69 

Semester 2 

11 (21.15%) 

18 (34.62%) 

12 (23.08%) 

11 (21.15%) 

52 


These ST (student) participants with multiple responses to individual assessment items, along with 
respective responses from educators (ED), generated 1878 instances. The key findings overall, 
related to the levels on Miller’s pyramid, are that: 

• In 397 (21.14%) of instances («=1878), students self-reported their performance at a level 
higher than their educators. 

• In 621 (33.07%) of instances («=1878), students’ self-reflection was in agreement with 
their educators’ perspective. 

• In 860 (45.79%) of instances («=1878), students self-reported their performance at a 
level lower than their educators. 

Despite a relatively low level of agreement, the scaffolded pattern outlined by students and 
educators in Figure 3 represents a consistent shift in focus from Knows/Knows how (acquisition of 
knowledge) to Shows how/Does (application of knowledge) for assessment across the 4 years of 
the course (p=0.84). Figure 3 also shows general agreement amongst students and educators with 
the exception of Fourth year students who reported that 39.2% of their assessments examined them 
at Knows/Knows how level, whereas their educators’ reported only 11.0% of assessments required 
their students to perform at this level; this difference was statistically significant (pO.OOl). 
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0 10 20 30 40 50 60 70 80 90 100 

Miller's pyramid (grouped) Knows/Knows How 


Figure 3. Educators’ (ED) interpretations of the curriculum contrasted with their students’ (ST), 
displaying a scaffolded assessment pattern from Knows/Knows how (acquisition of knowledge) to 
Shows how/Does (application of knowledge) over the four years of the program. 


Whilst this general picture is useful, the heat-maps (Figures 4a, 4b) provide richer insights into the 
students’ and their educators’ perspectives of the assessed and learnt curriculum broken down by 
NCS Domain, year of enrolment and each Miller’s level. The Domains and each Miller’s level 
which had greatest emphasis in the curriculum, that is, which were assessed multiple times, are 
depicted by “hot-spots” (the areas of darkest shading) on each heat-map. From the educators’ 12 
and the students’ 19 “hot-spots,” there were five areas of the curriculum where students’ and 
educators’ views aligned. These were: 


• Domain 2: Communication, collaboration, self-management (Shows how: Year 2). This 
shared “hot-spot” highlights that Second year students and their educators believed 
assessment required students to immediately perform at “Shows how” level. 

• Domain 3: Leadership and management (Knows: Year 3 and Does: Year 4). 

• Domain 5: Prepare pharmaceutical products (Shows how; Year 3). 

• Domain 8: Critical analysis, research and education (Does: Year 4). 

The data in Figure 3 suggested that the whole curriculum provides a clear knowledge foundation 
(particularly in Years 1 and 2), however deeper exploration of the findings (Figures 4a and 4b) 
reveals that some Domains appear to have stronger foundations than others. The number of 
occasions where Domains 3 and 8 were mapped to summative assessment tasks are shown in 
Table 2 and can supplement the heat-map findings. For example, according to the educators’ 
mapping on Domain 8 the assessments covered all three corresponding standards (8.1, 8.2, 8.3) on 
multiple occasions across all four Miller’s levels. As Table 2 shows, Domain 8 provides greater 
opportunities for student development. In contrast, according to educators’ mapping of summative 
assessment on Domain 3, students in Fourth year are expected to perform at a “Does” level with 
much less foundation development. 

The TLR data presented in Table 3 also provides an indication of the number of NCS the students 
were required to combine to successfully complete assessment tasks. Examined with the 
assessment description it is possible to determine if students were required to integrate the NCS 
(across and within Domains) at appropriate performance levels. For example, the Fourth year 
assessment task, a “Health Promotion Services Project Report,” required students to integrate 
Domain 2 (Communication, collaboration and self-management) with Domain 6 (Deliver primary 
preventative healthcare). Then, within Domain 6 itself students also combined Standards 6.1, 6.2 
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and 6.3 to deliver primary preventative care and were assessed by their educators at a “Does” 
level, that is, in the practice setting with clients. 

By interpreting the findings from Table 2 and 3 and Figures 3, 4a, and 4b, it is possible to 
appreciate (i) how the NCS were distributed throughout assessments across the curriculum, (ii) the 
number of times each Domain had been assessed, (iii) if each competency standard within each 
Domain was scaffolded throughout assessment from “Knows” through to “Does,” and, (iv) where 
exactly in the curriculum this took place. 

Feedback survey participants represented 45% («= 73), and 16% («=31) of those invited in 
Semester 1 and 2, respectively. Student feedback comments are included in the Discussion section 
to supplement the narrative. Interestingly, in Semester 1, Third year students had the poorest 
feedback survey response rate (30%) whilst Second year students represented the highest response 
rate (56%). In Semester 2, Second year students represented the lowest of all year groups (10%). 


Color Key 



0 02 0.4 0.6 0.8 1 

Value 



Figure 4a. Heat-map of students’ self-reflection of performance level on Miller’s pyramid for 

each NCS Domain by year. 
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Color Key 
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Figure 4b. Educators’ perspective; Heat Map of expected performance for summative 
assessment for each NCS Domain by year and Miller’s level. 
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Table 2. Distribution of competency standards within Domains 3 and 8, highlighting differences 
in scaffolding and number of times each standard was addressed in assessment by units and year 
of enrolment. 


NCS 

NCS Description 

Miller’s level 

# assessments 

corresponding to NCS 

# units corresponding 

to NCS 

Year of curriculum 

assessment conducted 

Domain 3. Leadership and Management 





3.1 

Provide leadership and organisational 

Does 

2 

2 

4 


planning 





3.2 

Manage and develop personnel 

Does 

1 

1 

4 

3.3 

Manage pharmacy infrastructure and 

Does 

1 

1 

4 


resources 





3.4 

Manage quality service delivery 

Does 

1 

1 

4 



Knows how 

1 

1 

4 

3.5 

Provide safe and secure work environment 

Knows 

2 

1 

3 

Domain 8. Critical analysis, research and education 





8 

Assessment covers all NCS in Domain 8; 

Does 

5 

1 

3 


8.1, 8.2, 8.3 



1 

4 



Shows how 

1 

1 

2 



Knows how 

2 

1 

2 



Knows 

2 

2 

2 

8.1 

Retrieve, analyse and synthesise information 

Does 

15 

2 

3 





4 

4 



Shows how 

16 

2 

1 





2 

2 





2 

3 





2 

4 



Knows how 

8 

2 

2 





3 

3 





1 

4 



Knows 

1 

1 

2 

8.2 

Engage in health, medicines or pharmacy 

Does 

8 

2 

3 


practice research 



2 

4 



Shows how 

4 

1 

2 



Knows how 

2 

1 

1 





1 

2 

8.3 

Formally educate and train students and 

Does 

6 

1 

3 


healthcare colleagues 









2 

4 
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Table 3. Two example assessments extracted from the TLR database highlight a First year 
assessment task requiring limited integration of Domains compared with a Fourth year assessment 
task with advanced integration across and within Domains. 


Example assessments 

NCS 

Domain 

Standards 

Millers 

First Year Unit'. 6 short in-class tests on 

1 

1 . 1.1 

Knows 

dosage form theory and calculations 

5 

5.1.3, 5.1.4 




5.2 

Shows how 

Fourth Year Unit: Health Promotion Service 

2 

2.1, 2.3, 2.6 

Does 

Project Report 


2.1.1, 2.1.3, 2.1.4 




2.3.1, 2.3.2, 2.3.3 




2.6.1, 2.6.2.2.6.3 



3 

3.1, 3.2, 3.3, 3.4 




3.1.2, 3.1.3, 3.1.5 




3.2.3 




3.3.1, 3.3.2 




3.4.1, 3.4.2, 3.4.3 



4 

4.2 




4.2.1, 4.2.2, 4.2.3 



6 

6.1, 6.2, 6.3 



7 

7.1, 7.2, 7.3 




7.2.1, 7.2.3, 7.2.4 



8 

8.1, 8.3 



Discussion 

Analysis of the TLR highlights two distinct viewpoints of an identical curriculum - those of a 
cohort of undergraduate pharmacy students and those of their educators. Whilst neither viewpoint 
should be considered more valid than the other, points of difference and similarity provide 
valuable insights and a meeting point for discussion of assessment and AoL. The value of these 
insights was supported by student comments from the feedback survey: 

• Provided useful feedback on particular aspects of the course. (Year 2, Femalel9) 

• It allowed me to see which areas (of curriculum) are relevant. (Year 3, Male21) 
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The authors’ claim the TLR is novel for two reasons - firstly, because it explicitly presented 
Miller’s pyramid to students for self-reflection purposes, which has not been done previously. 
Whilst Miller’s pyramid has been used for some time by educators to inform assessment plans 
(Wass, et al., 2001), in progress testing (Szilagyi, 2008) and curriculum review, and to discuss the 
difficulty (Epstein & Hundert, 2002) and scaffolding (Kleinert et al., 2015) of assessments, the 
literature does not appear to describe the explicit use of Miller’s pyramid with students for self¬ 
reflection. 

The second point of difference of the TLR was that it directly contrasted the student voice with 
educator curriculum mapping for AoL. The TLR supersedes traditional educator-led curriculum 
mapping focussed on the “intended” or “assessed” curriculum, which is not always the same as the 
“enacted” or “learnt” curriculum from the students’ perspective (Barrie, 2004; Edstrom, 2008; 
Harden, 2001; Lew, Alwis, & Schmidt, 2010; Porter & Smithson, 2001). Using this novel 
approach to AoL, the authors anticipated that the students’ self-reflection on the learnt curriculum 
would align with their educators’ mapping of the assessed curriculum, as found by other authors 
(Porter, Lloden, & Fuhrman, 1998, Plaza et al., 2007). However, the TLR did not report similar 
concordance, as evidenced by poor agreement between educators’ expected performance levels 
contrasted with their students’ reflections, especially when the curriculum was examined for each 
NCS Domain by year of enrolment. Kelley and Demb (2006) also reported disconnect in student 
and educators assessment expectations. The TLR may provide professional disciplines with a new 
solution to AoL. It provides a logical meeting point for students and their educators to ensure 
assessed curriculum translates to learnt curriculum. The meeting point is at the intersection of 
student voice with their educators’ curriculum mapping captured, reported and contrasted by the 
TLR. AoL principles impress that a curriculum which is designed with a solid foundation, 
scaffolded development and opportunities for students to integrate their knowledge, skills and 
attributes will enhance student success (AACSB International, 2013, Hall & Kro, 2006, Lawson, 
2015b). The TLR findings allow exploration of the design of this pharmacy curriculum against 
these three elements. 


Solid foundation 

Student success is enhanced if students are provided opportunities to learn foundational knowledge 
(Knows/Knows how) prior to being required to apply that knowledge in practice (Shows 
how/Does). Findings from this study, as seen in Figures 3, 4a, 4b and Table 2, illustrate that the 
curriculum provided students with opportunities to develop a strong foundation in some Domains 
but less so in others. This is particularly evident when contrasting Domains 3 and 8. Some of the 
unexpected patterns described, for example. Fourth year students’ disagreement with their 
educators may be due to conceptual misunderstandings of Miller’s pyramid, and the nature of 
knowledge, itself. This is illustrated by the following student comment: 

• I self-assessed at a higher level. I have worked in community pharmacy since first year 
and I feel/ have been told that I work very’ competently. (Year 3, Female21) 

Contrasting with this student’s statement, by definition, ‘Does” requires that the student or 
practitioner be situated in the practice setting integrating competencies to consistently perform a 
task. To perform that task safely, practitioners require a strong foundation in the underpinning 
knowledge of the skills (Miller, 1990; Wass, et al., 2001). A student may believe they are capable 
to “do” a task such as dispensing medicines but may be doing so without the foundational 
knowledge, namely, pharmacology and therapeutics, required to do so safely. 

With a specific focus on providing students with a solid foundation through the TLR assists 
educators to ascertain the AoL of an existent curriculum design. 
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Scaffolded development 

Outcomes-based educators argue for scaffolded development and multiple opportunities to assess 
competencies (Conway, Medina, Letassy, & Britton, 2011; Epstein & Hundert, 2002; Van Der 
Vleuten & Schuwirth, 2005; Wass, et al., 2001). Figure 3 portrays a curriculum which generally 
builds students’ capacity to apply their knowledge in practice by Fourth year. This scaffolded 
course development, described by both students and educators, is reassuring from a whole of 
curriculum point of view, however the differences between the students’ and their educators’ 
expectations at Domain level evidenced in the heat-maps warrants further exploration. The heat- 
maps may help us to understand how a scaffolded curriculum (represented by summative 
assessments) might translate to AoL. Through the eyes of students and educators the heat-maps 
provide evidence of Domains which appear well scaffolded and those which are not. The TLR data 
were able to demonstrate scaffolding or lack of scaffolding of summative assessments. 

The reported disconnect could be a result of differences in interpretation between educators and 
students or could highlight a need for curriculum restructure. The findings described may indicate 
some areas of poorly laid foundations, inconsistent scaffolding, or inadequate opportunities for 
students to integrate knowledge and skills: 

• There were many competency standards that I felt I wasn’t confident in despite the actual 
assessment showing that I was (Year 2, Male 19). 

The students’ heat-map on Domain 2 (communication, collaboration and self-management) 
implies students perceived assessment commencing at a “Shows how” level without assessment of 
underpinning knowledge development. If educators do not design curriculum to ensure scaffolded 
development from “Knows” (for example, the theory of motivational interviewing techniques 
utilised in smoking cessation) through to “Does” (for example, motivational interviewing with a 
client in practice) this could have negative results for those reliant on communicating with health 
graduates about their health and medicines. 

We now focus again on Domains 3 and 8 shown in Figure 4a, 4b and Table 2. Within Domain 8, it 
is apparent that a graduate has had their knowledge scaffolded and examined repeatedly to enable 
competent performance at a “Does” level for all standards, however they may be less prepared for 
performance in Domain 3. From this it is clear that Domain 3 (Leadership and management) may 
need to be introduced earlier in the course and scaffolded to ensure our graduates can be leaders of 
the profession. This example evidences the depth and breadth of information the TLR can provide 
for course review and AoL. 

To uphold AoL, students must be provided with clear instructions of assessment expectations and 
criteria in order to succeed (Biggs & Tang, 2007). Students’ difficulty in appreciating the 
relevance of the NCS may be a consequence of implicit use of the NCS by educators without 
explicit communication of their relevance to students. The TLR project highlights that there is a 
significant role for course level rubrics (Lawson, et al., 2015) or (if adopted) TLR rubrics to ensure 
consistent scaffolding across course assessment. Student understanding of educator expectations 
may be improved through students’ active participation in the rubric development phase. Rubrics 
may have addressed some of the difficulties experienced by students, as well as the evidenced 
disconnect, by ensuring students did not have to assume anything. One student offered that: 

I assume that we are being taught to meet the competency standards therefore by passing 
assessments I am demonstrating competency (Year 2, Female20). 
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Integrate knowledge, skills and attributes 

Pharmacists integrate multiple NCS to competently perform in practice at a “Does” level. To 
ensure workplace readiness, a curriculum must also be designed so that students are assessed for 
their ability to integrate their knowledge, skills and attributes. Harden and Stamper’s (1999) spiral 
design curriculum integrates relevant knowledge, skills and attributes and given the findings 
described here may be useful for future course redesign efforts. The benefits of integrated 
curriculum design and authentic assessment have also been discussed by Van Der Vleuten and 
Schuwirth (2005). The student feedback comments implied a science versus practice split 
indicative of potentially poor integration in the curriculum, particularly amongst First and Second 
year students. The relationship of the NCS to the curriculum was difficult for some students to 
appreciate and potentially led to students questioning the relevance of the NCS to their curriculum 
(and vice versa). This was evidenced in one student’s comment: 

The research is great, for me it has highlighted how truly meaningless the competency 
standards are at the moment. They let me know I’m “tracking ” towards competency, still 
meaningless ... I can only assume 3rd year placements will give gravity to the CS and 
finally allow me to judge myself and the School’s curriculum against them. ... Without the 
experience of real life practice (placements or intern) to benchmark the info against, the 
report doesn ’t have any meaning to me. 


(Year 2, Male31) 

Highlighted by the data in Table 3, authentic assessment choice can play an important role in 
providing students with opportunities to integrate their required competencies, be assessed 
formally and therefore succeed in practice. The Health Promotion Service Project Report which 
involves students on placement designing a health promotion service, delivering it to the 
community and evaluating it, required students to address multiple Domains and standards within 
each Domain. Whilst we cannot assume all the NCS listed were integrated by students, the data 
give us a positive indication that this was the educator’s intention. TLR data interpreted alongside 
assessment task descriptions confirm whether students were required to integrate their knowledge, 
skills and attributes to succeed in each summative assessment. This provides students about to 
enter the practice setting with confidence in their ability to perform. 

Overall, these findings have implications for students, educators and regulators; students may find 
it difficult to succeed if their expectations of curriculum are different to those teaching and 
assessing them and vice-versa. AoL could be sub-optimal especially as the learnt curriculum 
appears significantly different to the assessed curriculum at a Domain level. Course accreditation 
may be approved based on assessed/mapped curriculum which has been evidenced here to be 
different to the learnt curriculum. Overall, this may produce a mismatch in graduate knowledge 
and skills for their workplace requirements. 

Limitations and influencing factors 

To make use of the TLR findings for AoL it is necessary to be cognisant of three influencing 
factors affecting students’ and educators’ perceptions of their curriculum - student and educator 
familiarity with the NCS, familiarity with Miller’s pyramid and self-reflection difficulties. 

Poor NCS familiarity across the Australian pharmacy profession, including amongst its educators 
and students (Nash, et ah, 2015a; Nash, Chalmers, Stupans, & Brown, 2015b), may have 
negatively impacted on their ability to apply them in the education setting. As the NCS are action- 
orientated descriptions, the style of assessment and how well this matched the NCS and 
corresponding Miller’s description may have also influenced interpretations. Given the matching 
“hot-spots” emerged in instances where students were required to apply their knowledge (Shows 
how/Does), we can deduce it was easier for students and educators to appreciate the relevance of 
NCS when the assessment required application of knowledge, rather than acquisition of that 
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knowledge. Finally, some students reported confusion with the self-re flection task. This difficulty 
can be attributed to self-reflection being a new skill students had not previously encountered in the 
curriculum, compounded by the fact that students were required to recall multiple assessments that 
they had completed over the course of a semester and then identify relevant NCS for each. The 
following comments reflect this difficulty: 

• It was interesting seeing where the competency standards had been assessed this 
semester according to my lecturers - I hadn’t thought of all the different assessment 
pieces that were relevant. (Year 4, Female22) 

• ... useful in my current stage of study to just get the actual assessment because I was not 
really aware of where each of the competency standards were covered in my units. (Year 
2, Female 19) 

Furthermore, difficulties in separating a curriculum (as a whole ecosystem) and other activities 
taking place at the same time, for example, NCS flow charts introduced into unit outlines may 
have confounded results. Voluntary participation, student workload and survey fatigue may have 
led to low response rates. Participation most likely represented more motivated individuals which 
may bias results. 

Recommendations and future research 

Resourced appropriately, the TLR may scaffold student development of self-reflection skills. The 
TLR could also become a very powerful form of feedback to engage and empower students in 
their learning. It provides a central meeting point for educators and students regarding 
performance expectations for assessment, as well as offering insights on curriculum design for 
AoL. To ensure sustained engagement, future TLR versions should be embedded within courses 
and course teaching team cultures. In fact to maximise educational outcomes students and 
educators could workshop together to discuss the NCS, assessment expectations and co-design 
rubrics. If the NCS adequately reflect current pharmacy practice requirements, then the learnt 
curriculum as evidenced by students’ reflections must match the assessed curriculum. Student 
reflections therefore demand greater consideration in the curriculum review process. 
Internationally, professional courses could consider a similar approach to curriculum review, 
harnessing student interpretations to compare directly with their educators’ expectations for AoL. 

A project to determine the TLR’s impact on AoL amongst educators and students who meet to 
discuss their TLR is warranted, as are projects to explore the use of the TLR with other 
professional disciplines. 

Conclusion 

The TLR provides a new technique for evidencing the AoL of a curriculum. In the pharmacy 
course described, the TLR was able to highlight potential issues with foundation, scaffolding and 
integration of professional standards within a curriculum. The TLR reinforces the assessed and 
learnt curriculum may not always be consistent and provides educators and students with a logical 
meeting point for AoL. Using self-reflection, students can provide rich data reflective of a learnt 
curriculum useful for examining a course for AoL. This has potential portability to other 
disciplines with well-defined professional standards. 
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